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THE EXPERIMENTAL DETERMINATION OF STAND- 
ARDS IN FIRST- YEAR ALGEBRA 



H. O. RUGG 

School of Education, University of Chicago 



A report to the Mathematics Section of the Illinois High-School Con- 
ference of a preliminary investigation for the purpose of: (i) establishing 
certain methods by which standards for measuring the outcomes of a year's 
instruction in high-school algebra may be constructed; (2) stating tentative 
results obtained in eight Illinois high schools; (3) making certain criticisms 
of the learning and teaching process in algebra. 

The conduct of this investigation is based upon the following 
statement of the ultimate aim of first-year algebra: The successful 
use of algebraic symbolism in meeting new problematic situations, 
either, (1) purely algebraic situations; (2) general mathematical 
situations (such as are found in other mathematics courses other 
than algebra) ; (3) non-symbolic situations including various types 
of "practical" or "applied" problems not expressed in mathema- 
tical language. 

I. THE OUTCOMES OF FIRST- YEAR ALGEBRA 

Believing that some statement of the assumed outcomes of 
first-year algebra should be made the basis for further analysis, 
the following tentative statement of these outcomes is made : 

A. Immediate, specific, and preparatory outcomes. — These include 
the comprehension, interpretation, and manipulation of the specific 
"mechanical" operations involved in algebraic solutions; e.g., 
the four fundamentals as used with various type problems, factor- 
ing, removal of parentheses, etc. Certain of these are to be used 
in many specific algebra problems and in the solution of other 
mathematical problems; they are to be mastered as tools, prepara- 
tory to the taking of other mathematical courses, as well as for use 
as automatic tools for the solution of all types of "applied" prob- 
lems. These involve primarily the learning of rules — the formation 
of specific habits of manipulation. 
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B. Immediate generalized outcomes. — These involve recourse 
to selective, analytic, and conceptualizing abilities; ability to apply 
principles, in addition to ability to remember rules and to make 
certain fundamental habitual adjustments in the solution of general 
practical and applied problems and in solution of problems belong- 
ing primarily to other mathematical fields. 

C. Remote and less tangible outcomes. — The development of the 
ability to deal with general number concepts and "think quantita- 
tively"; the development of attitudes of (i) orientation in algebraic 
or general mathematical fields containing problematic situations; 
(2) confidence in one's ability to use successfully algebraic sym- 
bols in meeting new situations; (3) a broadened intellectual back- 
ground or perspective for the general cultural comprehension and 
interpretation of the scientific methods by which technical prob- 
lems may be solved, i.e., the development of a "scientific attitude." 
(In the preliminary attack on the general problem of standards in 
this field it is not proposed to consider this third group of outcomes. 
We believe standards in first-year algebra should deal primarily with 
the "mechanical" processes — i.e., those which should be made auto- 
matic — and only very little with these other more indefinite ones.) 

2. METHOD OF PROCEDURE 

At the 1913 meeting of the Mathematics Section of the Illinois 
High-School Conference a committee was instructed to begin the 
study of establishing standards by which the results of one year's 
high-school algebra might be measured. The problem, as attacked 
by the chairman of that committee (Mr. R. L. Modesitt of the East- 
ern Illinois Normal School), consisted of an attempt to determine 
the relative degree of difficulty inhering in a set of "mixed " algebra 
problems (i.e., a list containing both "mechanical" and "applied" 
problems), as shown by the varying proportions of a group of fifty 
pupils who were asked to solve the problems. This attempt, 
resulting in an adaptation of Thorndike's "mixed scale" (discussed 
below), was to be extended during the past school year. Other 
duties compelling Mr. Modesitt to give up the work, the conduct 
of the investigation was turned over to the present writer. Mr. 
E. L. Mayo kindly consented to assist in the work but has been 
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prevented from much active co-operation by unexpected and uncon- 
trollable circumstances arising in connection with the administra- 
tion of the classes in his school. He and his colleagues have 
helped, however, by giving the Thorndike tests in their classes. 
Thus the writer is alone responsible for the assumptions, line of 
attack, and present analysis and conclusions from the data. 

It should be emphasized in starting that we regard this investi- 
gation to date as of a preliminary nature only and not as one which 
has enabled us to state definite and final standards. The problem 
being turned over to the writer so late in the year caused the 
collection of data and organization of the work to be somewhat 
hurried. The material presented in this report is therefore more 
open to criticism of the "conditions" under which the data were 
secured than will be the case in the extension of the investigation 
during the coming year. We consider, however, that a sound begin- 
ning has been made on a very complex problem, the conclusion of 
which can come only in the course of years of careful and co- 
operative experimentation. 

The procedure since April, 1915, may be outlined as follows: 

1. Preliminary statement of the aims and the outcomes of the 
teaching of high-school algebra. 

2. Determination of basic method of designing and constructing 
tests for measuring efficiency in first-year algebra. (The investiga- 
tion of the validity of the assumptions underlying Thorndike's 
"mixed" scale, concerning (a) the relative difficulty of problems 
and (b) the validity of testing pupils with a scale composed of both 
"mechanical" and applied problems.) 

3. Classification of the subject-matter of first-year algebra and 
the determination of a list of specific operations whose efficiency 
should be tested. 

4. Determination of principles which should govern the selec- 
tion of test problems. 

5. Selection of problems composing each test with preliminary 
individual experimentation to standardize the arrangement and 
timing of these problems. 

6. Final organization of test sheets and giving of tests in eight 
school systems in Illinois according to standard directions. 
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7. The correction of the test papers; original tabulation of 
scores obtained; statistical treatment of results to state typical 
conditions; a minute analysis of the specific errors made in each 
problem. 

8. The interpretation of tabular and statistical material and 
conclusions concerning the progress of the experiment to date. 

3. GENERAL CLASSIFICATION OF SUBJECT-MATTER AND LIST OF 
SPECIFIC OPERATIONS TO BE MASTERED 

The subject-matter in first-year algebra is first roughly classi- 
fied to fit the assumed outcomes as follows : 

A. All material of a mechanical nature, necessitating complete 
automatism — the establishment of definite groups of habits through 
continued drill. The writer assumes that the following list of 
operations should be absolutely mastered as a result of the drill 
given in first-year algebra (there is stated in parentheses after each 
point the number of the test in which the efficiency of this operation 
is measured) : 

1. Addition and subtraction of positive and negative numbers. 

2. Law of signs in multiplication (Tests 1 and 2). 

3. Law of signs in division (omitted in design of tests). 

4. Addition and subtraction of monomials (Test 2; Test B-3). 

5. Addition of polynomials (omitted in design of tests). 

6. Removal of parentheses (Tests 1 and 2). 

7. Multiplication of monomials (Tests 1 and 2; Test B-17). 

8. Multiplication of polynomials (Tests 1 and 2). 

9. Multiplication of monomials by monomials, polynomials by poly- 
nomials, and monomials by polynomials (Tests 1 and 2) . 

10. Division of monomials, reduction of quotients (Test 3, problems 7, 
14, 21, 28). 

11. Use of exponents (Test 3). 

a) without parentheses: 

A. Multiplication of monomials involving 

1) integral numerical exponents (problems 1, 3, 8, 10, 15, 16, 
17, 19, 22, 23, 24). 

6) with parentheses (problems 4, 11, 18, 25, 27). 
c) negative integral exponents (problems 5 and 12). 

2) literal exponents (problems 2, 9, 16, 17). 

3) fractional exponents (problems 6, 13, 20, 26). 

B. Division of monomials involving exponents (problems 7, 14, 21, 28). 
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12. Division of polynomials (omitted in design of tests). 

13. Squaring binomials (omitted in design of tests). 

14. Substitution (Test 5; Test B, problems 1 and 18). 

15. Factoring monomials (Test 4, problems 2, 7, 12, 17, 22). 

16. Factoring of difference of two squares (Test 4, problems 1, 6, n, 16,21). 

17. Factoring trinomial squares (Test 4, problems, 4, 9, 14, 19, 24). 

18. Factoring trinomials of form ax'-\-bx-{-c (Test 8, problems 3, 8, 12, 
16, 20). 

19. Solution of equations of first degree with one unknown (Test 6; Test 

B-14). 

20. Solution of fractional equations (Test 7). 

21. Solution of quadratic equations (Test 8). 

B. All material of an original nature which, though necessitating 
constant use of definitely learned systems of habits, primarily 
involves independent thinking in applying fixed methods to new 
problematic situations: 

1. Applied problems leading to equations of first degree with one unknown 

(Test B, problems 4, 5, 9, 10, 11, 12, 16, 19, 21, 23, 25). 

2. Applied problems involving ratio and proportion (Test B, problems 

8, 13, IS, 22). 

3. Applied motion problems (Test B, problems 16 and 17). 

4. Applied problems involving translation of verbal expression into 

algebraic symbolism (nearly all of problems in Test B). 

It should be emphasized again that our attention has been 
centered primarily on the organization of the automatic operations 
and that the fist given here of the types of verbal problems is by 
no means considered exhaustive. Furthermore in connection with 
such topics as: the graph and its application, the formula, solution 
of equations in two unknowns, groups of three or more linear equa- 
tions, etc., we are far from a final decision as to their place in the 
course of study. It will be the aim of next year's study to establish 
a standard in this field. 

4. THE DETERMINATION OF A VALID METHOD FOR MEASURING 

EFFICIENCY IN THE LISTED OPERATIONS AND 

TYPES OF PROBLEMS 

The design of tests to measure efficiency in algebra implies a 
definite and detailed analysis of the specific operations and proces- 
ses to be mastered in the study of the subject. The design to be 
outlined here is based on the analysis made above. 
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A. Types of design. — There are at least three types of design. 
Type I. The Mixed Scale : Professor E. L. Thorndike assumes 

that one mixed test, containing "mechanical" problems of many 
types and " original " problems involving many sorts of fundamental 
operations, will be avalid measure of efficiency in elementary algebra. 

Type II: On the other hand, the opposite extreme view con- 
cerning algebra tests would have each specific operation tested 
separately; i.e., such elemental processes as addition of monomials 
and polynomials and the other fundamentals would be measured 
by separate tests. 

Type III : Between these two extremes it is possible to occupy 
a middle ground, by building a test-system which will test the larger 
and more important operations in a reasonably detailed manner, 
at the same time combining certain of the most elemental opera- 
tions in one test. The first type of design obviously cannot measure 
in any valid way many of the more important outcomes of the 
study of algebra. The second method, if carried to its logical 
extreme, would build up a test-system so detailed and cumbersome 
that no school system could hope to use it in the periodic determina- 
tion of pupil efficiency. Granted that the ideal "measuring-stick" 
would measure specifically every operation used in algebraic solu- 
tion, classroom and administrative requirements would prohibit 
its use. To get a usable measure, therefore, we must compromise 
in our desire for detailed analysis. The writer has imposed the 
criterion that the giving of the completed algebra standards to 
any class shall occupy not more than two or three class periods of 
45 minutes each. 

B. Fundamental criteria for the design of an algebra scale. — 
To measure algebraic abilities we must have a basis for determining 
the relative difficulty of problems and tests. There are four possible 
criteria: (I) the teacher- judgment basis; (II) "proportion-of- 
pupils-solving " basis; (III) the quantitative enumeration and 
objective analysis of the steps and operations involved in various 
types of problems; (IV) the solution of all types of tests by indi- 
viduals under carefully controlled conditions (i.e., individual 
laboratory tests) with detailed individual discussion and comments 
on particular problems and difficulties. The teacher-judgment 
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method assumes that the relative difficulty of algebraic problems 
and processes can be determined by the uncontrolled judgment 
of teachers of algebra. The second method assumes that the 
relative difficulty of problems will vary directly with the propor- 
tion of a large number of pupils able to solve the problems in ques- 
tion correctly. 

I. The Teacher- Judgment Basis: Implicit in the design of all 
of Professor E. L. Thorndike's "scales" (handwriting, drawing, 
composition, and algebra) is the assumption that "relative diffi- 
culty" or "relative merit" will vary as the proportion of "expert" 
judges varies who rank certain samples of student work as better or 
poorer than other samples. In order to apply the method to algebra 
he submitted 25 problems of various types (printed herewith in the 
order of their final ranking) to 200 teachers of algebra, asking them 
to "rank" the problems in order of difficulty. From the returns he 
selected a "scale" of nine problems, some of the automatic type 
and some of the applied type, some simple and some very complex, 
by taking the problems D, K, A, T, H, E, I, V, W, which an ap- 
proximately equal percentage of his group of judges had ranked as 
of successively greater degree of difficulty. That is, approximately 
80 per cent of 200 judges rank K more difficult than D; A more 
difficult than K, etc., throughout the list. Thorndike's deduction 
is, after some statistical manipulation, that the interval of difficulty 
between each two consecutive problems, D, K, A, etc., is equal 
and that these problems taken together represent a scale by which 
we may measure the efficiency of 20 weeks' instruction in elemen- 
tary algebra. (It should be said that Professor Thorndike states 
that a better scale could be designed by using other methods. We 
are criticizing here Thorndike's acceptance and continued use of 
this method, not his failure to recognize the greater validity of other 
methods.) 

LIST OF PROBLEMS USED BY PROFESSOR THORNDIKE 
IN DESIGNING ALGEBRA SCALE 

We state with each problem the percentage of pupils who solved the problem 

correctly 

(97.6) D. If <z=4 and 6=2 what does a+b equal? Answer 

(97.0) K. If <z=4 and b=o what does a+b equal? Answer 

(91 . 7) A. If x+$a=$a what does x equal ? Answer 
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(87.0) L. If 3*+4=2*+8 what does * equal? Answer 

(64. 5) 0. If 0=3 and b= 2 what does a 2 —ab equal ? Answer 

(62.8) S. If a=6 and b= 1 what does 2ab—ab 2 equal? Answer 

(36.1) T. Find the average midnight temperature for the week in which 

the daily midnight temperatures were 15, 3, o, — 7, —9, 6, and 

17 degrees. Answer 

(53. 2) P. If x— 2a-\-b= 2x+2b— 4a what does x equal? Answer 

x 
(61.0) U. If ~TT = a— b what does x equal? Answer 

(34.3) G. If a=6 and b=$ what does v aV 2b equal? Answer 

(53-9) B. The circumference of a circle=2nr. tr=3\. r=the length of 
the radius of the circle in question. If the diameter of a bicycle 
wheel is 28 inches, how many inches is the circumference? 
Answer 

. . _ T , 6*+7 2X— 1 , , ,„ , 

(19.0) C. If — =45 what does x equal? Answer 



(62.8) Q. If —^ — I — T~ — ~rr — ~T2 what does * equal? Answer 

x-\-2 x-\-$ x -\-$x-\-o 

(17.9) H. If = — 7 what does * equal? Answer 

Q> X cc 
X 

— I 

d 

( 8.3) E. If 2+ =0 what does x equal? Answer 

a 

(0.0) I. A man has a hours to spend riding with a friend. How far can 
they ride together, going out at the rate of b miles an hour and 
just covering the return trip at the rate of c miles an hour? 
Answer 

(3.5) R. Let e stand for the safe load that can be hoisted by a hemp rope. 
Let c stand for the circumference of a rope. If /=ioo c 2 for any 
hemp rope, how many pounds are a safe load for a hemp rope 2J 
inches in circumference ? Answer 

*+a * — a x 2 

M. If ; -=i what does x equal? Answer 

x—a x+a a'—x 2 n 

X. At what time between 6 and 6:30 o'clock are the hands of a watch 
at right angles to each other ? Answer 

.. T , a+b (x—a\i x—2a+b 

Y. If x= what does I 7) ; ; equal? Answer 

2 \x—b' x+a— 2b n 

J. If , . =~rr~ prove that a=c, or that a-\-b-\-c-\-d=o. 
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N. There are two thermometers or scales to measure temperature. 
The Fahrenheit scale (F.) is the one we commonly use. The other 
is called the Centigrade scale (C). A temperature of 32 degrees 
on the F. scale =0 degrees on the C. scale; 33.8 degrees on the 
F. scale = 1 degree on the C. scale; 35.6 degrees on F. = 2 degrees 
on C; 50 degrees on F.= io degrees on C; 14 degrees on F.= 
— 10 degrees on C. 

a) What on the C. scale=7o on the F. scale? Answer 

b) What on the C. scale=4 degrees below zero on the F. 
scale ? Answer 

c) What on the F. scale=20 degrees on the C. scale? 
Answer 

V. How much water must be added to a pint of "alcohol, 95 per cent 
pure" to make a solution of "alcohol, 40 per cent pure"? 
Answer 

F. A cube containing eight cubic inches was plated with copper. 
The difference in the weights of the cube before and after the 
plating was o. 139 lb. One cubic inch of copper weighs 0.315 lb. 
Form an equation from which the approximate thickness of the 
copper plating could be calculated. State whether the approxi- 
mate estimated thickness by your equation would be less or more 
than the exact thickness. 

W. Given that 2*— 3 is less than x+$ and that 11+2* is less than 
3*+S, to find the limits (i.e., the values) between which x lies. 

(It is understood that the pupil has not had any special training in inequalities 
or limits. This problem is, so to speak, an original exercise.) 

II. The Proportion-of -Pupils-Solving Basis: The foregoing 
basis (I) being open to criticism on analytical and psychological 
grounds, the writer reports herewith a slight beginning that he has 
made in investigating the validity of the method. The progress 
made has consisted of a comparison of Thorndike's results with 
those obtained by having students solve the same list of problems. 
His list of problems has been worked by 169 students in the Joliet 
Township High School, 92 of whom had just finished first-year 
algebra, 58 of whom had finished it one year before taking the tests, 
and 19 of whom had studied it two or more years before. The 
problems were arranged on the sheet in the exact order of difficulty 
as determined by Thorndike's rank method, giving what little 
influence the time element might exert to the support of his ranking. 
Table I below summarizes the number and percentage of this group 
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of students who solved each of the first 19 problems correctly. 
(They were given one class period of 45 minutes to solve the entire 
list of 25 problems. All attempted to solve more than the first 
19 and it is assumed here that the pupils had sufficient time to 
satisfy themselves that they could or could not work these 19 
problems. That is, we believe that the element of "pressure" 
or "drive" due to the "time" factor does not operate here.) 

What does the tabulation in Table I show? It shows that, 
instead of being solved successfully by gradually decreasing groups 
of pupils, the problems fall into a few sharply differentiated groups. 
Problems D, K, A, L, (Group A) can be solved by nine-tenths 
or more of the pupils who have had one year's algebra instruction; 
problems O, U, Q and possibly P and B seem to be about equally 
difficult, in general being solved by about two-thirds as many pupils 
as the problems of the first group; T and G are in another class 
of "difficulty" and C and H are in still another; E, I, R, M, X 
are again in a class by themselves, being so difficult to solve that 
practically no pupils can work them. Our tabulation confirms 
Thorndike only in the fact that the order of problems is the same, 
the interval of difficulty being decidedly unlike as determined 
by the two methods. To bring out this point more clearly note 
the size of the "interval of difficulty" between the seven problems 
on his scale and that determined approximately by the other basis. 

TABLE II 

Comparison op Relative Difference in Difficulty by 
Two Methods 



Relative Difference 
in Difficulty 
(Thorndike) 


Problems 


Percentage 

of Group 

Solving 


Approximate Relative 

Difference of Difficulty 

Based on Proportion 

of Pupils Solving 


idiff. 


D 


97.6 


idifi. 


2 " 


K 


97.O 


1 " 


3 " 


A 


91.7 


2 " 


4 " 


T 


36.1 


7 " 


S " 


H 


17.9 


9 " 


6 " 


E 


8-3 


10 " 


7 " 


I 


O.O 


11 " 



One outstanding fact appears: The judgment of teachers cannot 
be taken as a safe criterion for estimating the relative frequency 
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with which pupils can be expected to solve various types of prob- 
lems. The writer would go further and say that the judgments 
of teachers cannot be taken as a safe criterion in determining the 
relative difficulty of algebra problems, and that of the two methods 
thus far discussed the "proportion-of -pupils-solving" method is 
the more valid. Any method of determining difficulty of problem 
solution must be based on a sound and minute analysis of subject- 
matter and of the psychological processes involved in the solution. 
A most elementary analysis would reveal at once the distinction in 
difficulty- to- the-student, between the large group of "mechanical" 
operations which have been made more or less automatic by drill 
and the group of applied or original problems in which little or no 
drill has been given. That this analysis is necessary is shown by 
our tabulation, viz.: Practically all pupils can solve the simple 
automatic operations of substitution when expressed in "drill" 
form as in problems D, K, A; practically no pupils can use exactly 
the same processes when expressed in "original" form as in R. 
Approximately half of the students can solve fairly complex equa- 
tions of the first or second degree in one unknown when expressed 
in familiar "drill" form; practically no pupils can successfully 
use the same processes when needed in original or applied problems. 

III. Quantitative Analysis of Operations Involved in Prob- 
lems: As a further check on these two methods let us bring the 
third method of determining "relative difficulty" into review: 
namely, that of making a minute quantitative analysis of the prob- 
lems in question. This would consist of an enumeration of the like 
and unlike steps to be carried out by the pupil. To illustrate with 
the foregoing problems, we arrange the data as in Table III. 

The most casual inspection of the data in Table III is enough 
to lead to the conclusion that the relative difficulty of algebra 
problems to pupils cannot be determined by a mere quantitative 
enumeration of the number of like and unlike processes which 
the pupil must successfully handle. Problem R, a simple substi- 
tution problem, involving direct substitution of two quantities 
and one multiplication, is beyond the abilities of 96 . 5 per cent of 
these 169 pupils (presumably a normal group), while Q, a fractional 
equation including 1 factoring of a trinomial square, 5 multiplica- 



STANDARDS IN FIRST-YEAR ALGEBRA 



49 



tions, 2 additions, i subtraction, i transposition, i changing of sign 
(n operations in all), is solved successfully by two-thirds of the 
group. Clearly "difficulty" is not to be determined in terms of 
number of operations involved. 



TABLE III 



Prob- 
lem 


Process 


Number of 
Steps 


Substitutions 


Total Steps 


Percentage 
of Pupils 
Solving 


D 


Addition 


I 


2 


3 


67.6 






K 


Addition 


I 


2 


3 


97.O 






A 


Change signs 


I 
I 


I 


3 






Transposition 


91.7 






L 


Transposition 


2 
2 


2 


6 






Change signs 


87.0 











I 
I 
I 




3 












Subtraction 


64- s 






s 




I 
I 

3 


4 


9 






Squaring 






Multiplication 


46.1 








T 


Addition 


7 

i 




8 








36.1 






P 




3 
3 
3 




9 






Change signs 






Subtraction 


53-2 






U 


Recognition of squaring 

Multiplication 


2 

4 

i 




7 






Addition 










G 


Square root 


2 

I 


2 


5 






Multiplication 


34-2 






Q 


Factoring trinomial square. . . 
Multiplication 


I 

s 

2 

I 
I 
I 




ii 






Adding like terms 






Transposition 






Change signs 






Subtraction 


62.8 








R 


Squaring 


I 
I 


2 


4 






Multiplication 


3 5 







Thus we repeat, teacher-judgment of the relative difficulty of 
problems, even when aided by quantitative enumerations of like 
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and unlike processes, cannot be regarded as valid for the construc- 
tion of "problem-scales" of definitely evaluated units of difficulty. 
(From a detailed investigation of the problem in freehand lettering, 
the writer is also able to state that the teacher-judgment method 
of determining "merit" in samples of student work — unaided by an 
objective standard — is to be called into question. This leads him 
to suspect that results obtained by the methods in drawing and in 
composition will lead to the same unbalanced results as in algebra.) 
Obviously we have no satisfactory evidence that the seven problems 
of Thorndike's scale are separated by equal intervals of difficulty. 
In fact, from the point of view of the pupil who has to solve the 
problems, we know that they are not. 

This question of the evaluation of the bases upon which educa- 
tional standards are to be built is fundamental to the whole measur- 
ing movement. Many scales have been and are being constructed 
on the teacher-judgment basis in drawing, composition, handwrit- 
ing, algebra, and the sciences and applied sciences. Such a slight 
beginning as we have reported above indicates that the method is 
questionable. It is recognized that our data are somewhat meager 
— that a larger group must be used and a much more detailed analy- 
sis must be made. The writer is now carrying on an investigation of 
this question of "bases" as applied to various scales and will 
report results later. In the meantime we have cleared the way 
for the organization of our own procedure in establishing standards 
in algebra. It will be assumed in this preliminary investigation 
that practicable standard tests can be built on the hypothesis 
that, in a large group of pupils, relative difficulty will be approxi- 
mately indicated by the relative proportion of pupils correctly 
solving the test problems. At the same time it is specifically 
recognized that the final standardizing of algebra tests must include 
detailed consideration of a much more complete analysis. This 
leads us to speak briefly of the fourth method of evaluating "diffi- 
culty" in algebra problems, namely: 

IV. The Qualitative Psychological Analysis of Problem Solu- 
tion in connection with carefully controlled tests of individuals 
and complete introspective and interpretive data by the pupil. 
As a result of the first seven months of investigation of this problem 
the writer is convinced that no final standards can be determined 
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without a thorough utilization of this method. The learning process 
must be studied both objectively and introspectively with the 
individual student to reinforce and clarify the data secured from 
class testing. This latter may result in definite objective standards 
of the average number of problems "attempted" and "right" 
in a given unit of time for each specific test given. It will enable 
the teacher and superintendent to measure their school or system 
against the norm of many others. But it alone cannot be the sole 
criterion for the design and construction of the tests themselves. 

5. PRINCIPLES GOVERNING THE SELECTION OF TEST PROBLEMS AND 
CONDUCT OF TESTS 

In order that a scale for measuring efficiency in elementary 
algebra may successfully test automatic efficiency and independent 
solution it must be composed of two general types of test: (1) a 
specific test series (A) which will test the specific manipulative 
abilities of students in all the basic automatic operations involved 
in the solution of algebra problems; (2) a composite test (B) which 
will test the independent ability of the student in practical or 
applied problems. 

1. The Specific Test Series A, as designed and presented here- 
with, conforms to the following requirements: 

a) It is made up of a series of problem tests each of which is 
designed as a specific test for a definite automatic operation in 
algebra solution. 

b) Each specific test is made up of a number of problems (10 
to 28) each of an elemental nature and involving the operation 
in question, and each of approximately the same degree of diffi- 
culty (estimated here of course). 

c) Where it was impossible to arrange separate tests for all 
kinds of operations involved (owing to lack of time in classroom 
handling, etc.), those problems which involved closely related 
operations were grouped in one test and arranged in rotation. 
Thus the student solving 20 problems may be compared with the 
one solving 10 problems. 

d) Each test was designed as a time test, the time being so 
arranged (estimated) that no student could quite finish the test 
in the time given, but so that all could do a considerable number — 
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otherwise the measure of efficiency would have been too coarse. 
Care was taken to see that all pupils started and stopped the test 
at the same instant. 

e) The directions were all given orally by the experimenters 
so that differences in rate of reading and comprehending directions 
might not complicate results. 

/) Test problems were of the alternative sort wherever possible; 
i.e., they were designed to give either right or wrong answers — 
otherwise careful evaluation and weighing of answers would have 
been necessary. 

2. The Composite Test B. — An ideally designed composite test 
would be composed of many (say 25) "applied" algebraic problems 
varying in difficulty by approximately equal intervals and covering 
all the fundamental types of operations involved in algebraic 
solution. These problems should not include any of the specific 
problems of the first test series (A) ; they should be confined to the 
abilities of generalization, analysis, and application. Application 
of the rules and operations for which Test A is the immediate test 
is the primary function of the composite test. As indicated above, 
since we cannot use the teacher-judgment method or the quantita- 
tive enumeration of operations in this preliminary construction 
of the composite test, we are forced to the trial-and-error method 
of selecting a priori what seems to be a representative list of applied 
problems. The primary purpose of this study has been the stand- 
ardization of Test Series A and what has been done on Test Series 
B is of the nature only of a trial-and-error attempt to define the 
problem and a sound method of approach. 

On a basis of the foregoing principles Test Series A (8 specific 
tests) and Test Series B (composed of 25 applied problems) were 
constructed. We print below the first few problems of each " speci- 
fic " test and Test B complete. 

THE FIRST SIX PROBLEMS OF THE EIGHT TESTS IN SERIES A 
Test I Test II Test III 

(1) 5(4*-2)= (1) 3x+(x+i)= (1) a3. a s = 

(2) - 4 (3*-4)= (2) 4x+(x-2)= (2) a*-aP'= 

(3) -7(2 + 5*)= (3) » -(-7«+4)»= (3) 5a 7 -6os = 

(4) -3(5-8*)- (4) 2y-(y+ 3 )= (4) (» 2 ) 2 = 

(5) 6(-3*-5)= (5) 5*-(*-5)= (5) * _I * X = 

(6) -8(-4x-7)= (6) 3y(y-5)~y'= (6) (a 6 )i = 
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Test IV. Factoring 
(i) ^-64 = 

(2) 5**4-15* = 

(3) ax 2 +bx 2 +ay 2 +by 2 = 

(4) ^+4* +4= 

(5) (*+6)'-o= 

(6) tf-16 = 



Test VI. Solve for * 
(i) -I3X= 7 

(2) 4*+3 = 9a; — 6 

(3) 7*-5+ 2 * = I 3 

(4) 4+S(*-3)= 6 

(5) i8x =-31 

(6) *— 4=— 8^+14 



TestV 

(1) If a=3 and 6=2 what does a 2 —sab equal? 

(2) If c=6 and d=i what does 2cd—cd 2 equal? 

(3) If a=4 and 6=3 what does a 2 —ab 2 equal? 

(4) If e=5 and/=4 what does ef —2ef 2 equal? 

(5) If a=3 and 6=4 what does ab —20b 2 equal? 

(6) If x=4 and y=6 what does 2x 2 -\-xy equal ? 

Test VII. Solve for x 
Leave answer in form of fraction 

,-. 4*-2 _*~3 

3 ~ 4 

-6(2*-8) -3(2*+ 5) 2*+4 

(2) = 

w 5 4 10 



(3) 



x+i _5 
*— 1 3 



(6s+n) 
(4) 6x-s =13* 



(5) 
(6) 



3* — 4 X+2 

6 4 

5(^-7) _ 4(3^+6) _ -4(^-3) 
7 4 14 



Test VIII. Solve for * 
(i) * 2 -8l = 

(2) x(x— 2) = 

(3) ^+#x=6^ 

(4) X 2 —7X=—12 

(5) ^— 121=0 

(6) «(*+7)=o 



ALGEBRA TEST B 

1. If a boy is x years old, how old will he be in 5 years ? Answer 

2. An aeroplane that can fly 58.2 miles an hour in still air is retarded 
by a wind 9 . 7 miles an hour. At what rate does the aeroplane fly ? 
Answer 

3. If you represent a number by x, how will you represent 5 more than 4 times 
the number ? Answer 

4. Four increased by three times a certain number equals nineteen. Find 
the number. Answer 

5. Four years ago a man was seven times as old as his son and his son is now 
eight years old. Find the age of the father. Answer 

6. A train leaves Pittsburgh for the West at the same time that one leaves 
for the East. The former travels at the average rate of 42 miles an hour 
and the latter at the rate of 38 miles an hour. In how many hours will 
they be 240 miles apart ? Answer 
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7. Which increases more rapidly when r increases, the area of a circle or the 
circumference ? Answer Why ? 

8. A line 21 inches long on a certain map corresponds to 22 miles. A line 
75 inches long corresponds to how many miles ? Answer 

9. Eight times a certain number equals 45 diminished by the number. Find 
the number. Answer 

10. A father is 23 years older than his son and the sum of their ages is 49 years. 
How old is each ? Answer. Father is Son is 

11. The perimeter of a rectangle is 256 feet. It is three times as long as it 
is wide. Find its dimensions. Answer 

12. A can do a piece of work in 3 days and B in 4 days. In how many days 
can both do it working together ? Answer 

13. State whether the quantities mentioned below are directly or inversely 
proportional: 

a) The number of yards of a certain kind of silk and the total cost . Answer. 

b) Time a train needs to travel 10 miles and speed of train. Answer. . . . 

c) Length of a rectangle of constant width and area of rectangle. 
Answer 

d) Distance traveled by train moving at uniform rate and the time. 
Answer 

14. Given the formula a=-\ h(b—b'), find the formula for b'. Evaluate the 
result for 11 = 40; h=8; 6=6. Answer 

15. If a boy 4! feet tall casts a shadow 45 feet long at the same time that a 
school building casts a shadow 67J feet long, how high is the school build- 
ing ? Answer 

16. Find two consecutive numbers whose sum is 157. Answer 

17. If a train moves at the rate of r miles an hour how far will it move in t 
hours ? Answer 

18. The circumference of a circle is given by 2irr; ir is 3! and r is the radius 
of the circle. If the diameter of a bicycle wheel is 20 inches how many 
inches in the circumference ? Answer 

19. A post is i of its length in the ground, 5 of its length in water, and 9 feet 
above water. Find its length. Answer 

20. Find three consecutive numbers whose sum is 63. Answer 

21. A rectangular field is 10 yards wide and another is 12 yards wide. The 
second is 5 yards longer than the first and the sum of their areas is equal 
to 390 square yards. Find the length of each. Answer 

22. If a boy lying down, with his eye on the ground, sights over the top of a 
10-foot pole, held vertically 6| feet from his eye, and can just see over the 
top of a tree 373 feet from his eye, how high is the tree ? Answer 

23. A cistern can be filled with two pipes in m and n minutes respectively. 
In how many minutes can it be filled by the pipes together ? Answer .... 

24. The areas of two circles are proportional to the squares of their radii. If 
the radii of the two circles are to each other as 4 : 7 and the area of the 
smaller circle is 8 square inches, what is the area of the larger ? Answer. . 
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25. Find the number whose third and fourth parts added together make 14. 
Answer 

The following teachers kindly co-operated in the investigation 
by giving two class exercises to the work of testing their pupils 
and conducting the tests as a regular part of their school work 
according to our detailed directions: Miss Jessie Brackensiek, 
Quincy, 111.; Dr. E. H. Taylor and his teachers of mathematics 
in the State Normal School, Charleston, 111.; Mr. Ward Taylor 
and his teachers of mathematics in the State Normal School, 
Carbondale, 111. ; Miss Lida C. Martin and Mrs. Hostetler, Decatur, 
111.; Miss Fannie Andrews, Marshall, 111.; Mr. R. M. Ginnings, 
Macomb, 111.; Mr. O. R. Hedden, Robinson, 111. ; Mr. E. L. Moyer, 
Chicago Heights, 111. Our thanks are due to these teachers for 
their co-operation. It should be stated that at least fifteen 
teachers in other systems expressed their hearty interest in the 
problem but were prevented from co-operating only by the fact 
that their final examinations for the year were then being held. 
The reorganization of our investigation, coming as it did in the 
spring, necessitated this late request for assistance — otherwise 
a much larger amount of data would have been gathered. We 
have had, however, all the data that there was time to consider 
and more would have delayed the sending out of this report. 

Complete tests were taken by 518 pupils who were just finishing 
first-year algebra, giving us something over 4,500 test sheets in all. 
The treatment of these data may be outlined as follows: 

(1) Original tabulations: The tests were next corrected and 
tabulated in terms of the number of problems "attempted" and 
"right" for each pupil. 

(2) The average number of problems attempted and worked 
correctly in each test for each school, in one minute, and the 
averages for the entire group of pupils were next computed, by 
finding the harmonic mean. 1 For purposes of this prenrninary 

1 In this connection the reader's attention is called to the fact that in averaging 
data involving "time rates" it is necessary to use the harmonic mean, which is the 
arithmetic mean of the sum of the reciprocals of all the numbers. A statistical 
fallacy common to educational investigations involving time rates (e.g., see Starch, 
The Measurement of Efficiency in Reading, Writing, and Spelling, Madison, Wis., 1915) 
is to average such rates by the arithmetic mean. This practice invariably gives a 
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investigation these averages may be regarded as tentative standards 
against which to check the efficiency of the teaching process in any 
school. 

(3) The rank of each school was next computed for each test 
and the average rank of each school in all tests combined. 

(4) The relationship existing between the abilities involved in 
the different tests was computed by the Pearson product-moment 
method. 

(5) The percentage of problems attempted which were worked 
correctly in each test, i.e., the relationship between speed and 
accuracy in various types of test, was worked out. 

(6) The tabulation of particular problems correct in each test 
and a study of the validity of the method of constructing Test A 
was next taken up. 

(7) A detailed tabulation of the particular errors made by each 
of 100 pupils (selected at random) in each test. 

(8) A program of procedure for the continuation of the 
study. 

In order to accommodate the data of this study to the space of 
a magazine article they will be presented in tabular form with 
brief interpretation and discussion of each table. 

Table IV presents the typical features of our data for eight 
school systems. These data provide us with tentative standards 
only concerning the number of problems (consisting of definite 
types of operations) which should be worked in a given unit of 
time. Given a large and representative sampling of schools, we 
shall be able to refine our standard tests and their applicability 
to practical school measurement. For the time being the primary 
value of our results lies in the fact that they provide the basis for 
a critical revision of our tests, indicate the general line of approach 
in this problem of test construction, and lay bare certain weaknesses 
in the standardizing of the "teaching emphasis." These problems 
are fundamental, for the question of principles-of-design and 

result deviating from the true mean by as much as 15 per cent. The use of the arith- 
metic mean in averaging the time rates of this investigation results in an error exceed- 
ing that amount. Proof and illustrations of the effect of using the two means will be 
ound in the first of a series of discussions of "Statistical Fallacies in Educational 
Research," shortly to be published elsewhere by the present writer. 
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methods-of-construction must soon be worked out to a satisfactory 
conclusion. 

TABLE IV 

The Average Number of Problems per Minute "Attempted" and "Right" for Each op Eight 
Schools and for Entire Group op 518 Pupils 



Tests . . . 


I 


II 


m 


IV 


V 


VI 


VII 


VIII 


No. 
Minutes 

per 
Problem 
TestB 


School . . 


Att. 


Rt. 


Att. 


Rt. 


Att. 


Rt. 


Att. 


Rt. 


Att. 


Rt. 


Att. 


Rt. 


Att. 


Rt. 


Att. 


Rt. 


Att. 


Rt. 


A 

B 

C 

D 

E 

F 

G 


8.34 
7-05 

10.06 
8.39 
0-37 
9.91 
6. 54 

10.80 


6.60 
5-35 
8.00 
7.89 
9.16 
8.36 
4-45 
767 


3 
3 
3 
3 
4 
4 
4 
3 


45 
25 
37 
41 
37 
48 
41 
85 


1.38 
1.08 

LIS 
0.79 
2. II 
2.05 
I.24 
O.95 


7 
7 
8 
5 
4 
7 
6 
8 


50 
19 
57 
00 
95 
45 
89 
39 


4.30 
4.00 
4.72 
2.30 
305 
3.8s 
259 
2.67 


2.17 
2.70 
1-57 
2. 24 
1.85 
2.70 
2.21 
2.41 


1.38 
1.60 
0.79 
0.84 
1.08 
2.25 
131 
0.92 


2.17 
2.56 
2.71 
2.38 
244 
2.80 
2.63 
2.72 


1.27 
1.08 
0.98 
1.04 
1. 31 
1.63 
0.96 
1 .14 


2.8s 
2.30 
1.68 
1.60 
2.70 
317 
2.70 
338 


1.04 
0.92 
0.63 
0.97 
1.28 
138 
1. as 
1.35 


0.69 

• 57 

• 56 

■ 95 

■ 58 
.83 
.72 

0.85 


0.17 

• 19 

• 17 
.26 
.22 
.26 
.26 

0.21 


0.76 
138 
123 

L25 
O.87 
1. 45 


0.37 
65 
33 
.58 

• 41 

• 54 


2.45 
4.00 
2.65 

2.50 
297 
2.45 
3-19 


3-63 
6.20 
465 

3-65 
4-3* 


H 


0.99 


0.41 


5-45 


Average 
of 
schools 


8.30 


6.36 


3-92 


1.29 


7.06 


3-49 


2.26 


1. 19 


2-57 


1. 11 


2.50 


1.09 


075 


0.23 


I. II 


0.51 


299 


479 


Highest. . 
Lowest . . 


10.06 
6. 54 


9. 16 
4-45 


4.48 
3.25 


2. II 
0.79 


8.57 
4 95 


472 
2.30 


2.70 
1.57 


2.35 
0.79 


2.80 
2.17 


163 
0.96 


3.38 
1.60 


1.38 
0.63 


0.95 
0.56 


0.26 
0. 17 


1-45 
0.76 


0.65 
0.33 


4.00 
2-45 


6.20 
363 



Several outstanding facts may be set down: 

(1) Pupils can correctly solve five times as many problems of 
the type of Test I (simple removal of parentheses and changing 
of signs) as they can of Tests II, IV, V, VI — this in spite of the fact, 
for example, that the average number and kinds of steps and opera- 
tions necessary for solving each problem in the first two tests are 
almost the same, being for Test I, 3.5, and for Test II, 4. 2. 

(2) About the same number of problems can be solved per 
minute in Tests II, IV, V, and VI, again regardless of the kinds or 
number of operations used in each test. These data reinforce our 
conclusions that the relative difficulty of different types of problems 
cannot be obtained by a quantitative analysis of the problems 
concerned. Again, we have to date no known method of equating 
difficulty in different types of algebra problems. These points 
emphasize our contention that any standard test should be so 
designed as to test for one or at most two or three fundamental 
operations. 

Furthermore, the data given above bear directly on the question, 
To what extent is instruction in first-year algebra making habitual 
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certain "mechanical" processes represented by these tests? The 
answer is direct: To only a very slight extent. Instruction result- 
ing in a capacity for solving correctly less than one factoring prob- 
lem per minute cannot be said to be efficient, especially when one 
of the eight schools is securing a skill three times as great. It 
is the higher attainments represented by the work of these schools that 
are of significance to us, rather than the types, as they point out the 
possibility of practicable efficiency. This should be the primary 
function of the standard test. 

The blame cannot be laid solely to the door of classroom instruc- 
tion. Rather it should be laid on the teacher's organization of her 
classwork as expressed through her teaching emphasis. The 
figures given above show that no school is highly efficient or 
decidedly deficient in perfecting all kinds of processes. All schools 
are relatively efficient in the perfecting of some of these habitual 
processes. Clearly our results point to a great variability of 
teaching emphasis, some teachers spending their time making 
automatic certain operations, others laying the stress on other 
processes. Our data point to the need of a statement of the stand- 
ard amount of efficiency to be worked for by each teacher of algebra 
and of a more intensive study of methods of perfecting the particular 
manipulations. 

A seeming lack in the presentation of the foregoing data is 
some measure of variability. This will be expressed in the final 
standards in the form of the limits of the middle 50 per cent of the 
distribution. It has not been deemed expedient to take the time 
to compute these in this preliminary report. 

Our data thus point to lax methods of habit formation in the 
learning of these mechanical processes and to a decided lack of 
emphasis on ideals or attitudes of accuracy. But the most pertinent 
evidence which we have on these points is found in connection with 
an analysis of (1) the particular problems which pupils are able to 
solve in each test, and (2) the specific errors made by pupils in 
solving particular problems. 

It has been assumed that difficulty-to-the-student will be in- 
dicated approximately by the percentage of pupils correctly solving 
the problems in the tests. Careful study of the data in Table VI 
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will therefore aid us materially in the redesigning of our standard 
tests. It was intended (although not exactly carried out) to design 
the tests in accordance with the principle that the problems should 
be arranged in cycles, those problems necessitating exactly the 
same steps recurring every so many problems. This would enable 
us to compare the record of one pupil with that of another without 
having a specific test for every particular operation used. The 
data shown in Table VI for Tests I, III, IV, V, and VII in the main 
conform to this practice. That is, those problems utilizing the 
same specific operations are, in general, correctly solved by approxi- 
mately the same proportion of pupils. In the remaining tests, 
II, VI, VIII, however, while there is a rough paralleling of types-of- 
operations and proportion-of-pupils-solving, there is need for a 
much more detailed study of the problem. As a definite guide to 
the standardizing of the tests the data will be of great service. 
The study of the returns to date seem to indicate that the cycle 
principle must be adhered to strictly throughout the tests. The 
recurring problems must involve exactly the same steps in solution 
and the cycles must be of exactly the same size — otherwise the 
material will not offer comparable results. 

The standardizing of tests and of the teaching process will, 
furthermore, be much clarified by a detailed study of the particular 
errors which pupils make in solving these problems. Table VI 
and the descriptive list following present the data on this point. 

DESCRIPTION OF ERRORS ACCORDING TO NUMBER IN EACH TEST 

Test I 

No. of 

Error id 

Table VII 

i. Incorrect multiplication of figures. 

2 Failure to change signs with — sign outside parenthesis. 

3. Failure to use correctly with — sign inside parenthesis. 

4. Failure to follow directions. 

5. Error in writing. 

6. Used + sign inside parenthesis incorrectly. 

Test II 

1. Incorrect multiplication of figures. 

2. Failure to change signs with — sign outside parenthesis. 

3. Failure to use correctly with — sign inside parenthesis. 
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4. Failure to combine like terms or reduce to simplest form. 

5. Error in writing; omission of symbols, etc. 

6. Ignoring + or — sign and multiplying across them. 

7. Ignorance of operations needed. 

8. Addition instead of multiplication symbol following parenthesis. 

9. Omitted multiplication by symbol preceding or following parenthesis. 
10. Incorrect addition or subtraction of like terms. 

n. Omitted terms in the answer. 

12. Faulty multiplication of symbols. 

13. Failure to follow directions. 

Test III 

1. Incorrect multiplication of figures. 

2. Multiplied exponents instead of adding them. 

3. Added exponents instead of multiplying them. 

4. Divided exponents instead of subtracting them. 

5. Error in writing. 

6. Multiplied exponents instead of subtracting them. 

7. Multiplied denominator instead of numerator. 

8. Incorrect addition of exponents, e.g., gx m ^x=6ix" t instead of 6$x m + I . 

9. Regarding x— 1 as 1 instead of - . 

x 

10. Regarding y—'x 1 as xy (second powers canceling each other). 

11. (x°) 2 called x'; o power equivalent to first power. 

12. Subtracting exponents instead of multiplying them. 

13. General inability to use operation. 
14-19. Scattering errors. 

Test V 

1. Incorrect multiplication of figures. 

2. Incorrect addition or subtraction (— quantity the larger). 

3. Substitution of wrong figures. 

5. Error in writing. 

6. Incorrect addition or subtraction (+ quantity the larger). 

7. Incomplete. 

8. Inability to solve this type of problem. 

9. Incorrect substitution. 

10. Addition in place of multiplication. 
11-13. Scattering errors. 

Test VI 

2. In transposing terms neglected to change signs. 

5. Error in writing. 

6. Incorrect addition. 

9. Incomplete, not reduced to fractional form. 

11. Subtracted instead of added. 

13. Inability to handle operation. 

14. Multiplied across + or — signs before parenthesis. 

15. Incorrect division (inverted numerator and denominator). 
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Discussion of specific types of error — Test I: Over one-fourth 
of the errors made in this simple test may be ascribed to lack of 
fixing the multiplication tables in arithmetic. It is necessary to 
recall that Test I shows evidence of being fairly well automatized — 
the only test in the eight of which this can be said. Furthermore 
there being but few different algebraic processes involved (only two 
to five operations possible) the actual proportion of total errors 
found to be due to lack of automatization of fundamentals in arith- 
metic must necessarily be larger. The actual number of errors of 
any one type is small. Even so, each of the pupils on the average 
made two errors in solving twelve problems of this simple type. It 
is clear, however, that there is a need for more intensive drill on the 
process of removing parentheses, especially of the type of problem 
having the minus sign before the parenthesis. Table VII shows 
that the problems involving this operation, 6, 7, n, 19, etc., are 
the ones in which errors most frequently occur and that the problem 
is much more difficult to the student if a minus sign is used both 
inside and outside the parenthesis. The data confirm the view 
that teachers should analyze problems into constituent operations and 
drill on those that prove most difficult. The use of the minus sign 
is the stumbling-block in this most simple of algebraic operations. 

Test II: With a but slightly more complex problem and twice 
as many possible types of error we find in Test II a relatively small 
proportion of arithmetic errors. In 100 pupils there were found 
on the average four who made multiplication mistakes in solving 
each problem. The results of Test II confirm our interpretation 
of Test I that the use of the minus sign in removing parentheses is a 
source of weakness (22 . 2 per cent of errors were of type 2) and an 
element of instruction on which greater stress should be laid in 
drill. One-third of the nearly 600 errors made by 100 pupils in 
solving twelve problems are made in ignoring a plus or minus sign 
before parentheses (error No. 6) . The data show that this mistake 
is common to all of the eight schools but one. Clearly this is a 
type of error that should have been eliminated by class drill. To 
repeat: First-year algebra should make automatic the removal 
of parentheses. The evidence leads us to think that it is not doing 
so. If the data presented here are valid, 8 per cent of the 100 pupils 
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show distinct evidence of absolute inability to handle this operation. 
About 15 per cent of the errors may be ascribed specifically to 
carelessness in writing (omitting terms, etc.). Nearly all the errors 
that were made emphasize the careless and slipshod work done by 
our pupils, a condition due primarily to weaknesses in the teaching 
process. 

Test III : In the manipulation of exponents a few definite types 
of errors occur, viz.: Exponents are multiplied when they should 
be added and vice versa; problems of the type gx m - yx= ? are 
almost invariably solved as 63a:" 1 (error No. 8; another example 
of multiplying exponents instead of adding them?); the use of 
negative exponents has not been permanently fixed by at least 
half the pupils (errors Nos. 9 and 10) ; the use of the zero power has 
not been learned ; 24 per cent of all errors made point to an inability 
to handle exponents in general. The tabulations above emphasize 
again the need for analysis and sound distribution of teaching emphasis 
on the difficult operations. Complete mastery of the fundamental 
operations should be insisted upon. 

Test V: The surprisingly large number of errors made in this 
test bring out again the weakness in the mastery of the arithmetic 
fundamentals: on the average over three errors per person in solv- 
ing eight problems. Furthermore, one pupil in five made an error 
in addition, subtraction, or multiplication in solving eight problems. 
Further evidence of careless methods is seen in the fact that 20 per 
cent of the errors made are errors in substituting wrong numbers and 
20 per cent more are due to incomplete solution. As with the pre- 
ceding relatively simple operations so with simple substitution, 
many pupils are not able to handle successfully this process. 
Clearly, if elementary algebra should mechanize any "preparatory" 
operations that are used repeatedly in problems requiring independent 
thinking it should mechanize simple substitution. That it is not 
doing so is strongly suggested by our data. 

Test VI: In the solution of simple equations of the first degree 
with one unknown, eleven pupils out of a hundred neglect to change 
signs in transposing terms. Two-fifths of all errors made are of 
this type. The deduction from such data is clear: a primary 
purpose of elementary algebra should be to make automatically accurate 
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just such processes as changing signs in transposing terms. Teachers 
interested in improving the outcomes of instruction should use some 
such method for determining the difficult elements in the "learning 
process" in algebra, and redistribute their teaching emphasis in 
conformity to the difficulty of the operations in question. It is 
just such analyses as we are making here that will suggest to the 
teacher "relative difficulty" of operations and will point to a 
sound solution of the problem. 



SUMMARY OF CONCLUSIONS 

i. The subject-matter of first-year algebra should be definitely 
organized in the form of a specific statement of (a) the "mechani- 
cal" processes which should be drilled until perfectly habitualized; 
(b) the typical "original" or applied problems in which should be 
given at least a definite minimum of practice in the application of 
the mechanical processes to new problematic situations. (Such 
an organization is tentatively outlined above.) 

2. The efficiency of instruction in the mechanical processes 
may be tested by a series of time tests of the nature of those given 
herewith, each test being so designed as to test for one, or at most 
two or three, closely related processes. 

3. The efficiency of instruction in developing skill in the solu- 
tion of "original" problems may be measured by a standard scale 
of representative problems (the relative difficulty of each problem 
having been determined by some of the methods discussed here- 
with), the problems being separated by definitely determined 
intervals of difficulty. 

4. The study leads to the conclusion that the relative difficulty 
of the problems composing standard tests cannot be determined 
by the teacher-judgment method. There is evidence to support 
the view that difficulty-to-the-pupil will be indicated, at least 
approximately, by the proportion of a large group of pupils solving 
the problems in question. Difficulty of problems and relative 
"teaching emphasis" can be determined completely only by a 
detailed psychological analysis of each of the mental processes 
involved in the learning of algebra. 
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5. Standard tests, if representative of the conditions of algebra 
instruction generally, may be of definite service to the teacher and 
the supervisory officer by providing (1) objective measurement 
of the efficiency of instruction; (2) means of comparing efficiency 
with that of other representative teachers and systems; (3) means 
of determining particular weaknesses in the learning and teaching 
processes. 

6. The study of errors made by pupils indicates that inefficiency 
in algebraic solution is due primarily to lack of mastery (habitualiza- 
tion) of a few typical operations which recur frequently in such solu- 
tion. (Such operations include, e.g., the use of the minus sign 
in removing parentheses, principles concerning the addition and 
multiplication of exponents, the use of negative exponents, the 
use of the zero power, simple substitution, neglecting to change 
signs in transposing terms; etc.) This condition points to a need 
for a thorough study of (1) the psychology of the learning process 
in algebra; (2) the relative emphasis that should be placed on the 
teaching of certain processes, i.e., the relative drill emphasis. 



